add ckpt and restore with feature evict metaheader #4342
Conversation
✅ Deploy Preview for pytorch-fbgemm-docs ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Hi @lalala-2! Thank you for your pull request and welcome to our community. Action RequiredIn order to merge any pull request (code, docs, etc.), we require contributors to sign our Contributor License Agreement, and we don't seem to have one on file for you. ProcessIn order for us to review and merge your suggested changes, please sign at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need to sign the corporate CLA. Once the CLA is signed, our tooling will perform checks and validations. Afterwards, the pull request will be tagged with If you have received this in error or have any questions, please contact us at [email protected]. Thanks! |
|
Due to compilation issues with |
|
@q10 has imported this pull request. If you are a Meta employee, you can view this diff on Phabricator. |
|
Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Meta Open Source project. Thanks! |
|
Hi @lalala-2 I believe @bobbyliujb is working on this over at #4429 |
3 participants
1. Call get_block_size to determine the number of bytes each block contains, including the metaheader.
2. During checkpointing, call get_kv_with_metaheader_from_storage to retrieve the weights_with_metaheader for specified IDs. The first dimension of the weights_with_metaheader tensor should match the IDs, and the second dimension's number of bytes should match the byte size obtained in step 1. For example, if using uint8 type, the second dimension should match the byte size acquired in step 1.
3. During restoration, call set_kv_with_metaheader_to_storage to restore. Only need to provide weights_with_metaheader, as it contains complete information including keys, values, and feature counts.